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“Quality is free, but only to those who are 
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oF j Test cases: 
= Anti-Patterns: bet OMe 
e Only system testing a=-1;b=+1; 
e Testing only “happy paths” Pat fe 
e Forgetting to test “missing’ code uint16_t proc(uint16 ta, uint16 tb) 
return(result): 
; : } 
= Unit testing 
e Testa single subroutine/procedure/method Expected Test Results: 
— Use low level interface (“unit” = “code module”) aS, Dy ==> 6 


a=-1;b=+2; ==> 1 
e Test both based on structure and on functionality a 
— White box structural testing + Black box functional testing 
e This is the best way to catch boundary-based bugs 
— Much easier to find them here than in system testing 
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@ Carnegie 
} Black Box Testing Univer 
= Tests designed based on behavior bs 


e But without knowledge of implementation 
e “Functional” or behavioral testing 





= Test the what, but not the how 
e Example: cruise control black box test 
— Test operation at various speeds 
— BUT, no way to tell if special cases in code have been tested 
e Advantage: can be written only based on requirements or design 
e Disadvantage: difficult to exercise all code paths 





https://goo.gl/wJeZ56 


m Black box Unit Testing 
e Tests based on detailed design (statechart, flowchart) 
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m Tests designed with knowledge of software implementation 
e Often called “structural” testing 
e Sometimes: “glass box’ or “clear box” 





m Idea is to exercise software 
knowing how it is written 
e Example: cruise control white box test 


— Exercise every line of code 
» Tests that exercise both paths of every conditional branch statement 


— Test operation at every point in control loop lookup table 


e Advantage: helps getting high structural code coverage 
e Disadvantage: doesn't prompt coverage of “missing’ code 


— E.g., missing special case, missing exception handler Op ten Phinichcn cme 5 
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Coverage is a metric for how thorough testing Is 
= Function coverage 
e What fraction of functions have been tested? 
m Statement coverage 
e What fraction of code statements have been tested? 
— (Have you executed each line of code at least once?) 
= Branch coverage (also Path Coverage) 
e Have both true and false branch paths been exercised? 
e Includes, e.g., testing the false path for if (x) {... } 


= MCDC coverage (next slide) 





m= Getting to 100% coverage can be tricky 
— Error handlers for errors that aren't supposed to happen 
— Dead (unused) code that should be removed from source © 2020 Philip Koopman 6 
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= Modified Condition/Decision Coverage (MC/DC) 
e Used by DO-178 for critical aviation software testing 
e Exercise all ways to reach all the code 
— Each entry and exit point is invoked 
-— Each decision tries every possible outcome 
— Each condition in a decision generates all outcomes 


— Each condition in a decision is shown to independently 
affect the outcome of the decision 





https://www.youtube.com/watch?v=DivaWCNohdw 


e Forexample: “if (A ==3]||B==4)"” =» you need to test at least 
- A==3; Bl=4 (A causes branch, not masked by B) 
- A!l=3;B==4 (B causes branch, not masked by A) 
- A!l=3; B!l=4 (Fall-through case) 


- A==3;B==4 is NOT tested because it’s redundant (no new information gained) 


Might need trial & error test creation to generate 100% MCDC coverage 
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= Boundary tests: 
e At borders of behavioral changes 
e At borders of min & max values, counter rollover 
e Time crossings: hours, days, years, ... 
m= Exceptional values: 
e NULL, NaN, Inf, null string, ... 
e Undefined inputs, invalid inputs 
e Unusual events: leap year, DST change, ... 
= Justify your level of coverage 
e Trace to unit design 
e Get high code coverage 
e Define strategy for boundary & exception coverage © 2020 Philip Koopman 8 
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m Cunit as an example framework ne apes 


Test Suite: set of related test cases —— 
Test Case: A procedure that runs one or Suite '1' pees Suite 'N' 
more executions of a module for purpose of testing os OE 


Assertion: A statement that determines | | | | 
; " Test *°11°" ... Test ‘1M’ Test ‘NL’ ... Test ‘'NM' 
if a test has passed or failed 


http://cunit.sourceforge.net/doc/introduction.html 


= Testcase example: (http://cunit.sourceforge.net/doc/writing_tests.html#tests) 


int 


maxi( int 11, int i2) 


{ return (i1 > 12) ? 11 : 12; } 


void test_maxi (void) 
{ CU_ASSERT(maxi(0,2) == 2); // this is both a test case + assertion 


CU_ASSERT(maxi(0, - 2) == 0); 
CU _ASSERT(maxi(2,2) == 2); } 
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Use high coverage combination of white box & black box 
Use a unit testing framework 
— Multiple simple tests better than one huge, complex test 


Get good coverage of data values 
— Especially, validate all lookup table entries 


Creating test cases is a development effort 

— Code quality for test cases matters; test cases can have bugs! 
Difficult to test code can lead to dysfunctional “unit test” strategies 

— Breakpoint debugging is not an effective unit test strategy 

— Using Cunit to test 100K lines of code is not really unit testing 

Pure white box testing is “doomed to succeed” (neglects “missing” code) 

Don't substitute unit tests for peer reviews and static analysis © 2020 Philip Koopman 10 





https://goo.gl/SjzaB 


ee 
Your application is a special snowflake 


Excuses for 
Ifo) aa \\ Suletetoan Obebian Ronins 





O RLY? @ ThePracticalDev 


https://goo.gl/pvDMHX CC BY-NC 2.0 © 2020 Philip Koopman 11 


IMAGINE A LIGHT CONTROLLED 
BY THREE SWITCHES... 


WHAT’S MODIFIED CONDITION / 
DECISION COVERAGE TESTING? 


IN MC/DC TESTING, WE NEED 
TO SHOW THAT EACH LIGHT 
SWITCH CAN INDEPENDENTLY 


TURN THE LIGHT ON OR OFF... 


PHILIPPA 


HOW DOES THAT APPLY 
TO SOFTWARE? 


THE LIGHT CORRESPONDS TO 
THE DECISION AND THE 
SWITCHES CORRESPOND TO 
CONDITIONS 





www.rapitasystems.com 
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